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ABSTRACT 

Based on the need to develop an outline of the 
procedure's and' methods used in vocational education in Florida, which 
would then form the basis for communication/ discusision, : and 
development of a multi-State consorrtium for the production of 
performance objectives and criterion measures for occupational 
education, this report, one of five under a project to develop a 
multi-State consortium, reviewed four projects (developed j^n 
cooperation with the Florida Educational Research and Developmett 
Program) designed to develop pre-ob jectives, performance objectives,- 
and criterion-referenced measuring devices.. (The projects studied 
were,(1) Automotive Mechanics Assessments, (2) Producing Assessment 
Instruments in Ornamental Horticulture, . (3) Employability Skills 
Projects, and (U) Typewriting Commuaications. ) This report is divided 
into three parts: Part 1 is an introductory and procedufral / 
prese'ntation, part 2, the text of the fe^'ort^ is ad informational 
outline of the procedures and methods used by the Florida Research 
and Development Program to d-evelop performance objectives and- 
criterion-referenced measures- in occupational education The third or 
final part includes the appendixes related to ^the further development 
of a multi-State consortium. (HD) 
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'^is.' report is the result , of a study of four proiects underway 

andj'oj/recently completed in. the State of ^Florida Xv^hich were funded 

I through the Florida Educatlonal^P.esearch " and Development Prof»ram. 

• The projects vere designed to develop pre-objcctives , perforraancCi 
■ . , .... ' ^ ' f ' 

objectives, and criterion-^ref eren^ed mec jring, devices . The projects 

studied were as follows: , . ' 

(1) Autombtivp Mechanics Assessments, Grant Nu7h£>r S.D.E.' 
720-032 



.(2) • Producing Assessment Instruments in Ornaniental Horti- 
culture, Grant Nuinber S.D.E. 7K)-063 

6) Employabilitv Skills Project, Grant Number S.D.E.^" 
710-92 , . 

(4) Typewriting Communications, Grant Number S.D.E. 720^073 



The Souther^ Association of Colleaes and Schools,. Commission 
on Occupational. pucation Institutions, has entered a contract to 

organize a multij-state consortium to develop performance objectives 

"■c' ■ ■ i ' 

and criterion measures in Occupational Education. A part of this 
^oject was to develop a report outlining the procedures and methods 
used in Florida,:' which would form the basis for discussion with 
states int^ereste^d in the consortium. This report is organized so 
that, the text of the report (Part II) may stand on its own. /Th^ 
^report will provide a basic design for the proposed consort^uITv 
and appoint of departure for discussion and aevblopment. 
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INTPODL'CTTO N 

The purpoflCB of this study wer\^ to review four proVcctsi 
developed in cooperation with the Florida-. Fducntional Rcse<nr :h • 
and Development Prpgrara and to ]^roducc a report outlining the 
methods and procedures ui-;cd to produce pre-ob iectivos , performance 
objectives, and criterion- referenced measures in Occupational 
Education. This report will be utilized to communicate to other 
states the approach used in Florida, The rqport will fbrm the 
basis' of communication, discussion, and development of a multi- 
state consortium for the production of performance objectives ' ' 
and criterion measures for occupational educatipn. 

- During the process of the study, it was nojted that Robert F. 

MaR€r and his work in the area of developing and. evaluating 

♦ 

insti^ctional objectives had a great influence; upon the pi^cedures 
used by the Florida projects. This influence* is especially evident 
in the Request for Prbposals , published by the Florida Educational 
Research and Development Programs. 

Thl,s study is divided into three distinct parts. The first 
p^rt is an 'fntrodujptory ^alid procedural presentation-. The secondV 
pa'tt is the text of the reportY which is^signed to stand on its^ 
own as an informational report c^tlining the procedures^ and methods 
used l^y the Florida Research and Development' Program to develop 
perfoirmaqce objectives and criterion -referenced measures^in 
occupational education. The third or final part of the teport' 
wiU include the appendices and attachments, t ' ^ ' 



The pfimarv cortcept of th(> procediirys u5H*d hv, the Floritla 
•Researoh and -Devolopunent Pxoprnm was to dey^Mop n moans of making 
Inatractloa more account.able and to pivc it a basis upon wfiich. it 



mTght be better evaluat^-d,. revir,ed,* and -improved to more fully 
mecit th^e needs * of the* studertt and at the s.atr.e time the iob entry 
levtl requirernCfrits ofil)U3incss and Industry. , Procedures* uaed by 
th^ four pirojcct:5 had points, of vide variiauce, but the products 
prodiSced are renarkably similar'. This flexibility un*thc part of 
the Florida Educational Research and D^.-vc I^opniciiL Prof^.ram has 
produced the product and has», at the same ^ time, developed expertise 
of several people in identifying, writing, and. asse^ing performance \ 
objectlyes. ^/ " 
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METHOD OF pmc'rpnFK 

^ Tntervievs Conciiictoci ulxh FlVrlda Pcrspnnel 

Interviews were scheduled with ptrsonnel active in.tbe*^ 
adnlnistratiorK supervision, and conduct of tljt^c four projects. 
The ^rida/ Department of Education w\is visited on two occasions 
with interviews and ;t!iRcusGJ^ons taking place at. that level/ 
(See Appendix *'A'] for list of persons contacted and di^tes of 
contact.) Each primary 'investiRator was interviewed regarding 
the details of Ms project with particular emphasis on the 
procedural approaches and tnethods ol jjerf ormance objective 
development and criterion- referenced measures development. 
The process of validation was discussed, as well as the use 
of craft and advisory committee:^ and Che utilization of industrial 
and business persons other than craft and advisory members. Notes 
were made and compiled frow the ' Interviews . — In two of the four " 
projects, the coordinator was^ interviewed regarding his role in 

the project and'the details of rhe project activities. 

< 

In addition to the * primary lnvesti;>ators of each project, 
interviews were conducted with selected personnel in the Florlrda 
Educational Fe^arch and . Development Propran. The prlnary emphasis 
of these interviews was upon project processing, monltoring.'and ' 
dissenination. Proposal jleveloprcent and jthe project 
approval processes were also discussed, with -appropriate sanples 
of guidelines, forms, and other related riterial being furnished 
by the Florida Department of Education. 
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study of Prol'nct, Mflterialfi 

Rc#«arch of Needs: The Florida lulucatl cmal Research and 
Development Program established priorities for projects to be 
funded. Due to a very active legislature, top pr^iority is being 
•given to making?: programs of education both more effective and 
more accounC«il)lc . A weakness was' that it was difficult to 
assess the Instructional- effectiveness based upon rather general 
objectives being used by most teachers. To -make Instruction 
more accountable/ high priority was given to the funding of 
projects which would develop performgfite objectives and 
criterion-referenced test items designed to measure in specific 
^ terms the instructional process. 

Guidelines and Fdrms: An extensive guide for project develop- 
ment was prepared by the Florida Educational Research and Develop- 
ment Program^ which provided background information, approval 
processes^ project management and control, payment conditions 
etc. In relation to this report, probab«ly the most pertinent 
information was the definition of Performance Obj^ectives and 
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Criterion- Referenced Test Exercises listed In Technical Spe41f Icatlons 

\ . . 1 

fet Catalogs of Objectives and Assessment Items, publication. 

These specifications are appended to each contralf't involvinjr the 

production of objectives. A listing of these specif icatipns is . 

reproduced in Part II of this document. ' v ' 



Technical Specifications for Catalogs of Objectives and 
Assessment Items; Florida EMucatiotjal Research and Development 
Program, Florida Department ; o^ Education, Knott Building, 
Tallahassee • Florida. 

li - 



tliich phase ot th*' lour pro)*'rt.s, u-hitMi itu-liulvMl ;to 1 i r 1 L at Ion 
of project propos.jls, iiu»int)r.iniluni (>( .i^'.n'^'n^^iH , "Stato o[ th<» Art^* 
reports, and tlu» tinl.shi^J produrts of cl»reL' ot ilio tour pro-JectH, 
was carefully rovi<*wod. Tlw* final rtij^ort <ind product of the 
Typewriting Coinnuni cat I ons projrcts were .not c<Jniplt»Lc, hu^a- dcaft 
of each was furnished by the lUireau of Vocational Kirsoarch and * 
Kvaluation of the Florida r)eparLmer)t of Education for the purpose 
of this study. • 

Durlnr. the final stapes of the interview process, the Florida 
Fducatlonal Pcscnrch and Dovelonnent Pro^^ram Staff recommended the 
uae of the Automotive Mechanics Proiect as a model for outlining? 
and pfQsentlnj; the Florida approach to developing performance 
objectives and criterion-referenced measurements. The descriptions, 
accounts, and prpcedures presented tn Part II of this report are 
presented in terms of t\\e Automotive Mechanics Project for the 
following reasons: 

(1) The project was thp last ^of the proup of four projects 
•covered by this report to be funded,. Since it yas the 
last. It had t»he advantage of inco rporatlng . the findings 
of the first three projects, irhich seemed to contribute 
to a more rapid development- of p^re-ob j ectlves , performance 
objectives, and criterion-- referenceG measures. 
- (2) The project was developed in the s>iortest period of time. 
(3) The total 'cost of the project was lower than the former, 
projects . ' ' 
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•These acj^antages ,can be attributed partly to the development 
^^^^ 

•of expertise on the part of th^ Florida Educational Research 
I and Deve:p|)ti]ient Program and the experiences ' of earlier investi|;ators* 
Workington the three previous projects in occupational education. 
Other feethods. and techniques used by the investigator of the 
Automotive Mechanics Project could have contributed to the apparent - 
efficiency of the project. ^ 



Florida; Educational Research and Development Program has definite 



reporting''''ktid product submitcal dates designed Into the contract. 



. Project Management . • ' 

■"^^ ^ This secti^on 6f the report covers the management processes 
used at the Florida State Department of Educ^lon^ l^^?e^ and^ those 
management te^chnlques which are used in the field wi;ve^e"' research 
is being conducted* Project monitoring and review techniques 
are discussed as they are presently being conducted by the Florida 
ducatlonal Research and Development P'rogram*^^ 



Methods Used to Develop Performance Objectives 

In this part of the'^rQport, the method used to develop 
pre-objectlves and performance objectives is explained; A 
careful review of the four projects studied for this report 
resulted in an approach which s'eems to have valid support from 
within the projects and from the craft and advisory committees 
used at various times during the projects* 



Methods Used to Develop Criterion-Referenced Measures 



The projects reviewed In this study developed pre-objectives, 
performance objectives, and criterion-referenced measurement 
devices simultajj^ously . Although they Were accomplished the - 
same prpject,, there were some definite differences in the tech- 
niques used. Because of these differences, this report treats" 
them independently. 
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Validation Procedures - . , 

: ' ■\ . ' ' -mr 

■ This section of the report e>rplaij¥s the methods used to 

^ * / 

^. - ^ /// *^ 

validate the perfarrmance object Ives ^;;Jand crlterioiv-ref erenclfd 
measures. The Invplvem^nt. of general advisory coiranlftee and 
craft committee members 1b exulalned, as 'W^ll ^s the field , * , 
testing design. / 

Process and Product Evaluation • 

The process and product evaluation is 'designed into each 

project. The use of crafc and advisory committees, technical j 

consultants and advisors, and the involvement 'of a Department 

of Educatlon;fPoor;dlnator provides on-going product and process 

evaluation. A^vetwork of participating field- test schools, 

coupled with cost analysis information, provides f6r accountability 

^ - ■ 

and applicability of the project products. 



Dissemination of Materials 



/' • 



Elements in this portion of the report deal with the strategy 
to^ be considered for dissemination of the catalog of' pre-objectives, 
performance objectives, and criterion-referenced measurement 
devices. The present "design of dissemination is explained. 
Modlflcatl^ons are presently under further^ study and analysis . 

In-Service Training of Instructional and Supervisory Personnel 
The proposed plan for in-servipe training of instructional 
.personnel ,and supervisory personriel is contained in this section 
i)f the report. The dissemination of catalogs and their proposed 
use in staite assessment, instructional course development, and 



Istudent assessment efforts are outlined. This* portion .of the 
report. Is based primarily . ..on ideas gathered during the interview 
portion of the study and on those portions covering in-service ^ 
training*^ which were a parX: of the ">J)Emorandum and Agreement." 

Hethod of Revision 'arid Updating oy the Catalogs 

'■ . - • I- 

The nature of technological change- in modern society 

requires that all materials developed for occupational education 

have as a part of the developmental process rieans by which these 

materials ci?r. ue revised and updated. This section of the report 

presents a- proposed reyision and updating cy^le which would perml 

I' ■ . , 

periodic revision through the ^nvolvement of content specialists. 



ins truptlonal personnel, superyl^MMersonnel, and craft 
committees. / • » • 
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AN APPROACH TO DEVELOPING PERFORMANCE OBJECTIVES AND 
CRITERION-REFERENCED TEST EXERCISES IN OCCUPATIONAL EDUCATION 



PREFACE • . 

This part of the study contains the text of the report. It 
Is designed tpj communicate to educators, arid particularly to 
occupational educators, a process ased bv the Florida Educational 
Research and Develpptnent Program for the production of * pre-obj actives 
(preliminary performance objectives), performance objectives, and, 
criterion-referenced measuring devices. In order to 'keep the 
report 'as brief as possible-, the outliri^ method of presentatibn 
wa$ selected. More detailed explanations are available through 
the office of tl^e Commission on Occupational Education Institutions, 
Southed Association of Colleges and Schoois, 795 Pcachtree 'Street; 

N. E., Atlanta, Georgia 30308, - 

■ - ' . «" 

»■ ' ^ 

^ ' . ' . -> 

The approach outlihed is' oha method of achieving the production^ 

of catalogs of performance objectives and criterion-referenced* ' 

■ ■ •■ • ' ■■• . 

measures %nd, thus, should not be considered a. rigid or inffexibl^ 
plan which cannot be revised or changed. Its primary purpose is 
to serveiilas a point of depai;'ture for discussing the proposed , 
operation and in^plement^ation of the multi-state consortium. A" * 
detailed process and format will be developed by the consortimnJ 




members if the idea becom^^c% reality. . 
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A. DEFINITION OF TERMS . / * • 

1. Prc-objectlve: A -pre-objective is statement in 

measurable terms of an observable behavior to be exhibited by 

the learner. It possesses each of tfie following elements or 

characteristics: ^ •* 

a. Situation /This situation confronting the learner 
^ Is clearJLy stated. / " 
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^b. Action - The action required of the learned is 
identified. ■ , , 

' • . c. Object - The object , on which the learner Is to 

operate (I.e., thd object of the ^ . 
* * • action) is clearly stated. 

d. ^Limits -^e limits associated with the activity 
^. ' ■ ^expected of the learner are staped._ 

^Measurablllty . r An observable performance Is 

generated by the stated action. 

2. Performance Objective:- A Performance objective is <a 

statement In precise, measurable terms of a particular behavior' 

to be exhibited by the learner under specified conditions. It 

^possesses each of the elements or. characteristics specified 

below: . 

a. Situation ~ The situation confronting the IrC^Kxer Is 

clearly specified. Including the m'odi^ 
,,ln which stimuli are to be presented. 

b. Actldil - The acfcJLou required of the learner is 

unamblgously defined, Including phe . 
mode 1ft Which responses are to be m^de'; 



c. Object - The object on which the learner Is to^ 
operate (I.e., the object of the action) 
Is clearly stated. 

19 . • , 



/ . d. Limits - The particular limits associated with - . 

. , 1* . thB activity expected of the learner are 

specified- (Limits may be placed op , 
situation, Action and/or object.). . , 

e. Measurability - The specified action is an observable 

rather than an iijf erred response. 

f. Communicability - The ob^^ctive is^o stated that one, 

» and only one, interpretation 

of the objective is reasonably - 
' ' possible. 

r. - • . 

g. Criterion -'The deip-ree of proficiency required as 

' \^ '■ - ' eyidence of accomplish men^by a student gf 
' . . '\be objective is' indicatecT? (The criterion^ 

' ^ ■ may '?ie» indicated ^Implicitly or ^explici^tly 

. \ If iinplicit, . 100% accuracy, is effectively . 

^ • desi^natefi."»Tf. explicit, may be Appended 

parenthetically^ to the statement of the . 

, - ' I objective.) ^ ^ 

• *3. eriterion-Referenced* Test Exerciser A criterion-referericed^ 

test exexcis6 is an exercisie based upon a^lprf ormance objective 

is designed to allw -the determination of whether o^ not the learner 

>' '''^ ' » ^ ' ■ ' , ■ 

Has a-ccpmplished t4ie objective. ;-J[.fr"^ssessets each'of the character- 

tstics specified below: ' \ ^ ' ■ ^ ^ / 



a. ' CongrUen.ce - TRe task specified in the item corresponds 

\^ dir^tly to the performance specified in 

the objectiv-e, including 'the situation, 
• I \ . action, object, and limits. 

b. Compreh^nsibility r- The item-specified task is so stated 

i or portrayed thiat the learner ^ 

clearly underg^nds wbat is expected 
of- him. '^^A 

c* Objectivity - The exercise ^(including component items, 
' ' if ^ any) is stated in such away that all 

competent observers (evaluators) can make 
a clear and unequivocal decision as to 
' whether or not the learner h^ demonstrated 

an acceptable performance. 
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*d. Integrity - The exercise is structured in such a* way • 

that an acceptable response to the exercise 
constitutes sufficient evidence, in and 
of itself, that the learnet has accbmpllshed 
the corresponding objective. 
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-e. Equlvalerice - If two or nore exercises correspond * 
^ * . ^ to a simple objective, each exercise 

in the set V7©uld be a true* lLj.temate, 
^ • . iTi that a student who passes for • * 

** - fails) one exercise on a given occasion 

would be expected to pass (or fail) 
. . • anv other exercisein the set. 

* ^ k\ D^ain of Interest: The total content covered by a subject 

or occupatic^. Domain charts, as they become a part of a ta'&k 

analysis, provide the limits within which the per forman^ie objectives 

and criterion-referenced test exercises are developed. 

. . ^ - ^ \ ' * 

. B. . REQUEST FOR PROPOSALS _ ' * . ^ V " ^ • ^. 

This document; identifies ideas which j:ieed' research' and develop- 
ment and serves as a means of solicl|^ng proposals from public and 
privatf organizations, agencies, and institutions which have the ^ 
abllit}^, personnel, and\technical competence to complete the project. 
The document provides the philosophical, fiscal, and administrative 
•strategies utaPtzed to insure successful compjjetion of project. - 

The Request for Proposals outlfnes^the suggested format of 

' V ■ 

the proposal including major headings such as Introduction. 
Discussion of the /problem. The Approach, and The Work Plan. 'Also 
in9luded are topics fully explaining the management procedures to 
be used in the^.project, such as general organizat;ional charts, ^ - 
facilities to b^. utiMzed, description of related experiences of • 
the brgdniiation, agency, or institution, sub-contracting plan 
(if applicable), and the method of project control (reporting 
periods).. Each proposal is developed with descriptions of personnel, 
travel, supplies and materials, and other costs. Specific amounts 



are shown , for each cost with Che rationale f6r determining the 
cost -included as a part of the description. { 

The Request f or^ - Propogal^ include^ guidelines for J>roject ^ 
format, methods of eval^uation, plan for *contrcict awards, and general 



contractual provisions 



DEVELOPMENT OF AGREEMgNTS 
Upon approval of the proposal, the successful proposer develop 
with the Florida Lducational Rese3kxb)'and Development Progr^ni a 
Memorandum of Agreefinent^. This Memorandum of Agreement sets forth , 
the obligation of each party' and establisrfes reporting and product 
delivery dates. Also explained in the Memorandum, of Agreement are 
other special conditions such as schedulles for payment, copies of 
-trie product required, person (s) to whom project personnel are to 
report, any special validation requirements, dissemination of 
materials, and in-service training of personnel to be users of the . 
product. This Memorandum of Agreement becomes the legal binder 
of thi^ two parties to the conditions set forth .in the proposal. 



D, REPORTING PROCEDURES 

Periods of reporting ere a, part of all Memorandum of Agreement 
and are mutually established by the successful proposal writer and 
the Florida Educational Research and Development Program. Reporti^ 
requirements are established on specific dates and at various 
intervals through the life of the project. Reports take the form 
of progress reports, rough 'draft^of the product, or the finished 
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product itself. 



Reports and/or drafts the product are coordinated directly with 
payment schedules. <When the specified report or product is delivered, 
'under conditions .of the Memorandum of Agreement a pre-determined 
amount of money i$ pai»d. ^ ■ ^, 

E. PROJECT MANAGEMENT . >^ . . 

A project monitor: is assigned to each project upon the develop- 
ment and -approval of the Memorandum oi Agreement. *This person(s) 
interprets the ^contract to. the project personnel and serves as 
a reporter to the Florida Educational Research and.yf)evelopment 
Program regarding the progress of the project. In projects involv- 
ing production of objectives and tesy exercises, he is a ^ubject- 
matteu specialist and approves or^disapproves the product (output 
with respect to content. Thq^guide for monit^orltig- is the^r'Schedule 
of Events" required of (the cbnTr^tor. This monitor also makes 
recommendations for payment as the ne«tessary reports and/or' products 

are delivered and accepted. All projects calling for the expenditure 

-. . ■ . . « 

of amounts in excess of $15,000" require that requests, for proposals 
be solicited. Thus completion of an earlier stage of product 
development xloes hot guarantee^ continuation .by the same contractor. 

■ • : 

Each project involving productioa of objectives is assigned a 
technical consultant, whose functions include training of project 
personnel in writing techniques, review of materials produced with 
respect to technical quality, dnd final judgement with respect to 
the technical adequacy. of the objectives and 'test; exercises. 

In addition to the monitoring and reporting features required 

' "v ' • ■ 



. ^ .1 ' ' 

by the Florida Educatlonal -tHescarch and DevTelopnient ^P-rpgram, the 
* policy of annual contracting with provisio'ns that no contract may 
extend beyond a fiscal year provides further safeguards. 

' F, METHODS USED TO DEVELOP PERF0RMA::C£ OJ^JECTIVES 

1. Study 9f the ;*State of the Art": 

■ A comprehensive study is made of all material related 

to the curriculum area under development • linformation is obtained 

from other states,^from business and industry, and in some case^, 

from foreign sources.' The material is studied in terms cxf its 

applicability to the domain being developed and the quality of 

the material being reviewed. Exhaustive efforts are made through 

the ERIC files *to identify and study items of research which 
i,. 

* seems to be- related to the domain. In some cases, questionnaires 
and surveys are sent out to collect the needed and/or 'additional 
information. The Florida Education Research and Development Program 
strongly urges a thorough and comprehensive study of the "State 
of the^rt" before beginning an independent development of perfor- 
mance objectives. i . ^ 

A» In addition to public and private occupational 

V . — ■ '1 

education programs, prt^ject personnel , are encourageii t^^urvey 

industrial training - programs conducted by companies themselves. 

.In certain areas, ^ study of the, military training materials provide 

.pertinent and applicable performance objecf Ives and criterion- 

referenced .measures and should definitely »be considered in the "State 
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o^ the Art" study 
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2. Domain Chartln'g for the Occupacignal Area Being Developed; 

A chart is developed which identifies the domain*'or 
instructional territory over which perfontanc^ oMective? and 



criterion-referenced measuires* are to be developed. This chart is 
usually designed to provide all performances expected of workers / 
in a particular occupational category. From witAin this seK are 
selected those performances which are judged by employers and" 
craft committees to be entrv-level skills. In the broadest 
sense ^tTilsj could be applied to the Cotal domain of automotive 
mecjvanics and to lesser domains within the broad classification. 

An example of a lesser instructional domain tinder automotive 

J - 

mechanic could be automotive tune-up specialist. 

* * The domain is identified by\^c use of the Dictioniry 
of Occupational Titles, by craft committees numbers who *arp employed 
and a^'ively working in the occupation being developed^ and by 
conducting a task analysis of a broad trade area where the occij^tion 
being developed is broadly represented. The domain Is then 

^tablished through congruence of the three sources of informatic^. 

The domain chart culminates in rather specific statements defining 

competencies. These compeffencies are, in turn, defined by a given 

set of performances stated in objective form. 

3.* Development of Pre-Objectives: 

In working toward a perf ormancerbased instructional 
program, a complete set of pre-obj ecti ves are developed. The term 
pre-objective Is merely <;horthand for "preliminary performance 
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objective." Since the criteria for T^fe-ot^ject^ves are less stringent 
and because the. resultant task description is relatively brief, 
contractors are able to define the dbmain in terms of performance 
with relative ease- Wher^ pertinent, partl-cfularly in the case 
of job-entry skills, each pre-objectivt5^ expande^to the level , 

of specificity requir^r4 4^ ^ per f ormance bbj ect ive . The^pre- 
objectives provide the 'framework fSr present programs desiring to 
move toward a per^orma'ftce-based course and new programs which 
are to be started. ' , ^ 

A. Use of Content Specialists: 

Content specialists are employed as writers of both 
clasa^ of objectives and Vhe test measures. Their knowledge 
and expertise provides the professional guidance In th^ wording,, 
comprehensiveness, and applicability of the performance objectives # 
These persons are aided and frequently trained in writing techniqiies 
^bythe technical consultant (s) assigned to the project. 

5. Use of General Advisory Committers; 

The general advisory committee is appointed early 
in the project and assists in guiding the investigator through a 
planfled studv of -the domain and the processes of business and 
' industry as they. apply to the domain. Its function is primarily 
one of advice and counsel through the project. It provides the 
investigator a mirror for his ideas, his procedures, and the develop- 
ment of the finished product. 
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^; 6. Use of Craft Advisory Committees: 

This committee is of vital imnortance* because of 
the responsibilities it is assi^rVied. It is madp up of persons 
who are adtivelv emploved in the occupational ;^rea beinp developed; 
'and, in addition, it has as members outstanding teachers in the 
occupational area beinr (developed. Thev have as their chief 
resDonslbllitv the review of the nre-obi ectives , performance 
objectives, and criterion-referenced test exercises. The. :vise 
with respect to the annropriateness , the utility, and the comprehen- 
siveness of all objectives and measures developed by any method in 
the project and pass on all materials developed by the content 

specialists. It is their responsibility to identify the businesses 
k 

and industries which will be involved in the validation procedure. 
They assist in the development of surveys , interview formats, 
and questionnaires' to be used In screening Industry and obtaining 
their feedback. The proper selection, orj2;anization, and utilization 
of this committee is of prime importance in the vali<|ptlon of the 
materials produced by the project. 

7. Catalo^inp Pre-Objectives , Performance Objectives, and 
Criterion-Referenced Test Exercises: 

Each project produces a catalog which orjjanizes the 
pre-objectives, performance objectives, and criterion-referenced 
test exercises in a logical order for use by teachers and super- 
visors. A codinp system is developed which is used to organize, 
classify, and assist in the rapid retrieval of the pre-objectives, 
performance objectives, and criterion-referenced test exercises. 



2A 

The system identifies the type of objective or measurement Involved," 
general subject area, specific occupational area, classification of 
th% objective or measure, grade level of material, and the tvpe of 
learning process Involved (discrimination, verbal, p^oble^Nsolving, 




cognitive, etc.)* Within the domain, a numeric coding s^Sxem 
identifies the pre-ob jectives and performance objectives included 
for each specific oegrtent of the domain. The criterion-referenced 
test exercises af^ located with t*fe"'pba:^ormance objectives and 

ordinarily require no additional retrieval. 

■ i 
8. Field Testing of Materials: ^ 

A part of each project approved by the Florida Educational 

r^h and Development .Program is the field testing of materials. 

Definite events are identified which must take plsbce during field— 

test activities. These events are an* integral part of the validation 

process used in each project. The field toting includes use of 

test consultants, identification of test sites, types of students 

to be used in field tests, methods of testing, involvement of 

teachers and supervisors, collection of student Information on 

data sheet (thos^ Students participating), and a report of the total 

field test results. 

G. METHODS USED TO DEVELOP CRITERION-REFERENCED TEST EXERCISES 

The. most common production sequence employed in the process 
of creatllljjjl^ ^llfct test measures is (1) pre-objective, 

(2) trial criterion measure,] (3) performance objective, and (4) 
revised or final criterion measure. The process of stating in 
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testable form ttie ta8k(s) to he performed in the accomplishment 

of the pre-ob1ectlve serve*? to' clarify the Ah.lectivr and facilitates 

the more complete specification required for the correspond inp 

performance obiective. v>iln those cases where development to the 

"performance objective" level is not required, the practice has 

been to provide a sample (Proto-tvpe) item rather than a complete 

exercise. The criterion-referenced test exercises are ordinarily 

developed; validated, and field tested at the same time as the 

pre-objectives and performance objectives. It is the function 

of the technical consultant assip;ned to the project to advise, ^ 

r^iew, and approve the test it^ms or exercises developed by a 

project with respect to measurement aanects. 
i 

H. VALIDATION PKOCEDimE<r 

Validation is an integral part of the total Process of product^ 
development. The catalogs of pre-ob jectives , performance objectives, 
and criterion-referenced test exercises are develop«jd- by content 
specialists and test consultants who are recognized as leaders in 
their field. Their work is then processed through general advisory 
committee men^bers, craft^^ committee members, project staff personnel , 
and finally field tested* under actual instructional conditions. 
Each process of validation' has specific criteria by which the 
pfoduct is evaluated before it is permitted to be developed to its 
next s^age. involvement in the validation process is extensive 
and comprehensive. Of special significance ^s the use of business 
and industrial practitioners who are either working in the field 
or supervising -workers in the field*. They are surveyed, interviewed, 
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and, in some Instances, Ftc.rve on craft committees during the process 

» 

of each project In occupati oijal education. 

I . DISSEMINATION OF MATPPTAT .S 

Dissemination of the catalogs Is partially covered as a part 
of the Memorandum of Agreement* A specific system for dissemination 
Is not yet fullv Implemented or spelled out. Each project director 
has certain Ideas regarding dissemination Including who should be 
respMjjJjllble for this part of the* total task. As a result of this 
study* the folilowlng comments are made which'could lead to a well 
developed and svstematlc dissemination plan. The general consensus 
seems to favor a dissemination plan x^^hlch would permit the flow of 
produces from the projects to the Florida Educational Research 

\and Development Program and, thence, to the various sub-dlvlslons In 

y - f \ 

the Department of Education. From these sub-dlvlslons, the materials 

/ ' 

would be distributed In 'Connection with well planned In-servlce 

/ ■ 

training programs for supervisors and teachers* During the Interview 

t . - J- ' ' 

process, the general consensus was that the best person to do this 

would be that state level person havlAg responsibility for a service 

area (T & I, Agriculture, D, E», Office Occupations, Etc.)» It 

was the consensusj' of those Interviewed that all personnel who are ' 

to use or supervise the use of the materials be thoroughly trained 

In the implementation of -the performaj^ce-^ased materials and the 

methods of evaluating their effectiveness. ^ 

J. IN-SET^VICE TPAJNTNft OF PEKSONMEL 

A vital step toward the implementation of a performance-based 
curriculum is an In-service and pre-servlce teacher education program. 

30 
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Such a progratn-will ^^evelop skills on the part of existing and new 
•teaching and s|pervisor personnel in th<i' utilization and evaluation" 
of performance-based instruction. The implementation of the ^ 
perforwance-based curriculum renuire55 _,an orientation to this • 
method as well as 9;p|ecific instruction in how to select and utilize 
pre-bbiectives, performance objectives, and criterion-referenced 
test exercises. There are at least two fields of thourht-repardlnR 
who should develop performance objectives and the tole of the 
teacher in the development and implementation of a performance- 
based curriculum. 

One of the current trends identified by this study 1^ that 
most tochers should be educated in skills necessary to properly 
select and organize pre-obj ectives , performance objectives, and 
criterion-referenced test exercises which have already been ' 

developed by the procedure outlined in this report. They would make 

i 

their selection from a comprehensive; and complete catalog of 

performance objectives and design th'elr materials accordingly. 

I 

Another point of. view holds that all teachers should be , 
trained to vrrite performance ob,1ect|'ives and criterion-referenced 
test exercises using the catalogs ff pre-objectives as a broad 
frame of reference. T'ith this point of view, the ouestion of 
validity and expertise enters thejpicture, as does the difficulty 
of educating all teachers to prepare these objectives. 

The plan suftgested by this report takes the former, point of 
view — that most teachers could be educated-more efficiently. 



31 



with less In-servicc training coRtf5, nnd v.-l th a hlpV^er dppree of 
validity and reliability to skillfully Rclert and orpanlze 
performance- oMectlves and criterion-referenced test exercises ■■ 
from a catalop. 

K MFT Hnns USED TO REVISE AND UPDATE THE CATALOGS OF PKE-OBJECrivES 
PERFORMANCE OBJECTIVES. AND CRITER ION-REFF-RKNCED TEST EXERCISES , 

Today's technological advances are takinp place at a rapid 

rate. This is characterized by the chanpe of models which occur 

in evervthinp from automobiles to hlph-spe°ed comnnters. A program 

of-atudv, oarticularlv one which is nerf oi*anl^-based , must have as 

a definite elei^ent a means bv vThich materials, nractices, and procedures 

can be continualJv reviewed, revised, and updated. 

In keepinp with this frar^e of reference, the followlnp^ four 
elements are necessary in order to keep performance-based materials 
current and up-to-date. Each of the elements can he expanded and 
elaborated upon but are presented in an unconnlicated form for 
the purposes of this study. 

/ . 

(1) In-service ?nd Pre-service Training 

This activity would introduce materials to the teachers 
and supervisors.' They would be trained in the skillis of selecting 
appropriate performance objectives and criterion-referenced test 
exercises from a comnrehenslve catalog. This would permit a 
teacher to enter the cvcle at anvtime thev are selected to implement 
a performance-haspd curriculum or at the time they are selected 
for their" initial nerlod of emnlovment. 
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(2) Submittal to Local Craft CoTrmlttcee 

After one vear of use, the teacher would condbct Craft 
Advisory Committee meetings (at th*e;local level) to determine the 
appropriateness of the performance objectives and criterion-referenced 
test exerciser and their applicahilitv^to the local employer needs. 

(3) Sta^e-wide Jn-service >^eetinp 

The third vear of the^cvcle would involve a- meeting of 
all teachers in th^ir occupational areas to review and make 
recommendations for changes in the master catalog of performance 
objectives and . cri terion-ref erenced test exercises. ^ 

(4) Contract for Updating the Catalog • - 
k 

5very fourth vear, a contract would be axi/arded to mpdate 
the catalog. This process would follow basically the same procedure 
used in developing the initial catalog. 

Upon completion of these four steps, the- cycle would start 
over. A period of third partv evaluation should occur at the end 
of each four-vear cycle to determine the effectiveness of each 
phase, the degree of involvement, and the nualitv of the revisions 
made through the cvcle,, Other components of evaluation could be 

■ ■ ' J 

Included at this time. The chart on the next page illustrates the 
plan for revising and updating the catalogs. 



PART III 




ERIC 



36 



APPENDIX A' 
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PERSONS INTERVIEWED 



NAME 



TITLR 




Dr. Roy Glehls, Jr. 
Dr- K. M. Eaddy 

Miss Lucy C. Robinson 

Miss Gall Trapnell 

Mr. R. L. Gamble ^ 
Mr. John Healy 

< 

Dr. Jimmy Crews 
Dr. Jim Hensel 
Dr. 'Glen Shlnn 
Mr. William F. Wleser 
Dr. Wallace Hannura 
Dr. J^m Selman 



Consultant for Evaluation 
State Department of Education 

Chief, Bureau Vocational Research 

and Evaluation 
'State Department of Education 

Assistant Administrator 

Business Education 

State Department of Education 

Assistant Administrator 

Distributive Education 

State Department of Education . 

Consultant, Industrial Education 
' , State Department of Education 

Coordinator -for Research and 

Development 
State Department of Education^. 

Project Director 
Typewriter Communications 
University of Florida 

Head, Department of Vocational- 
Technical Education 
Unl veracity of Florida 

J?roj«ct Director ^ 
Ornamental Horticulture 
University of Florida 

Project Director \ 
Automotive Mechanics Project 
University of South Florida 



11-17-72 



11-17-72 



11-17-72 



11-21-72 



11-20-72 



U-20,.72 



11-15-72 & 11-20-72 



12-29-72* 



Project Director, Employablllty Skills 12-28-72 

FlorTda A &. M 

Florida State University 



Project Coordinator ' ' 
Automotive Mechanics Project 
University of South Florida 



12-29-72 



EKLC 
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APPENDIX n ^ 



^ PROGRESS HEPORT ON THE PROJECT 

January 12, 1973 ^ 



PART I - S tudy of the "State of the Art '.'. ^ 

* 

The study of the "State of the Art" ha^ been in progress since 
November 15^ 1972. To date, an ERIC search of, documents has been 
completed. A computer search has been made recently to include 
materials added since the ERIC search was conducted. Microfiche 
copies have been made- of all available material through the National 
ERIC System. . ' 

search for materials wt^icK migh.|!^|)e-^iippropriate -has beeryjiffd^ 
in the Educational Resources Index and the International Dissertation 
Files. Three books written by Robert F. M.ager have been purchased 
and are being read f pr^'Tresource information-. * ^ 

In. addition to** the above items, a questionnaire and letter of 

>*_'■* , ^ 

introduction of the project vas mailed to the eleven states making 

up the Southern Assoqiation of Colleges and Schools plus Missouri, 

Oklahoma, and Ohio. The questionnaire and letter was designed to 

assess the current status of performance-abased instruction and 

criterion- referenced measurement in. the states and to determine il 

the states were interested in the possibili ty. of * forming a consortium. 

As of the date of this report, nine states have. replied to the 

questionnaire, and all nine have expressed an affirmative reply 

and have asked for more information about the proposed consortium. 

9 
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In addition to t)ie written rcaponaofl, two other Ktateo have • 
verhnllv cxprcRwod an InteroRt in heconlnr Involved in the 

development. One state has offered possible fimdinr, aaslntancc. 

Contact is beinr made v^th that 'state to determine their Intercnta 

and funding participation. 

* • * . 

A detailed review of all literature collected anil computer 

seailch materials is continuing. Some consultation assistance is 

beinp used in the orj>anization of the"" materials and in editing the 

written materials produced by the project. For a detailed review 

of^ the plans for the cominp two months, please refer to Appendix C, 

Calendar of Activities. 

PART A - Outline of Procedures Used to .Develop Performance 

Objectives and Criterion-Referenced Test Exercises 

As of the submittal of this progress report, th±^ portion of 
the contract will have beenVcompleted with the exception of editing 
changes to be made by the Florida Educational Research and Develop- 
ment Program. 

PART III - Development of Proposed Agreement for the States Forming 
the Consortium 

A sedrch of the ERIC file has been made and appropriate microfiche 
obtained of material havipg to do with the development of consortia 
and educational cooperatives. A computer search to update the material 
was requested and was delivered January 11, 1973. A study of the agree- 
ment forms and organizational structures will result from the research 
of these materials. (See Appendix C, Calendar of ActivlMes, for 
further detail •) 
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Interview schedules have been rst.'il>ll shed tor vl«itinjp ench 
of the states expressing nn afflrmntlve inlererU In the prolect. 
Contact persons have been identified in each of the interested 
States and permission has been rcccUvcd to contact the key 
curriculum personnel and the State Directors of Vocational 
Education, Interviews will bcp.ln on January Id, 1973. and continue* 
throuj^h March 7. 1973. (Sec Appendix talendar of Activities, for 
details). 

PART V - Orpanlzlitlon and Mnn.^mcnt of a Conference of Proposed 
Consortium Members 

Preliminary planning has begun on the timing of the conference 
and the settlnp of tarret dates for certain events to take place 
leading to the conference. Details of the conferenx:e should be 
complete by February 15. 1973. and will be submitted with the next 
progress report. (See Appendix C. Calendar of Activities, for 
events leading to the conference). 

PAFT VI - Final Report of the Project 

Little work has been accomplished at this time except for project 

X . 

achievements to date which will contribute to the final report* 

<# 

The project director has discussed the proposed consortium with 
a large number of persons who are leaders In Vocational Education, 
These informal discussions have taken place during travels for . 
the project, in meetings ^htild by the Commission on Occupational 
Education Institutions, the American Vocational Association Convention, 
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and the Annual. Meeting of the Southern ^.ssociation of Colleges 
and Schools. Without exception, the idea has been well received 
and. in many cases, persons have offered t6 beccine actively 
involved if their services are needed. This has been especially 
encouraging, to the project director. 



41 



4 




s 





Umm OF ACTIVITIES - BEN A HIPST, JR. 



APPENDIX C 



wn 





MOAY 


TUESDAY 


WEDNESDAY 


THUKDAY 


FfllDAY 


%mtm''' 




/ 

NEWyEAfi'SDAr 


f\ Work On 
*^ Research Of 
Documents At 
R.C.U.-Knoxvllle 


Q-Uork On 
^ Research Of 
Documents R,C,U. 
KnoxvlUe. 
Registration for 
Dissertation 
Winter Quarter. 


h Work in COEI 

Office on 
rough draft pf 
Florida Project 
First Report. 


^ Work on 
Florida , 
Project COEI 
Office. 


6 


> 


^ Finalize 

First Peport 
On Florida 
Project. 


p Finalize 
First RpDort 
on Florida 
Project. 


Work pn Florida 
Project Researcf' 
Materials (ni- 
croflnns) 

finish Draft of. 

1st Pnt. on Fla. 


//work on Fla. 
Project GOEI 
Office. 

Prolect 


13. Mail First 
Report on 
Florida Project 

r 


13 


'i 

(' 

C 

i — - 


f5 Vork In rOF 
Office on Pe- 
search Material 

fnr Pin T'rft^p/>^ 

^elf-Studv 
Meetinp. 


7(J- i''ork On 
"Stjite of the Ai 
Report" 


n 

t , 

i 


IS 

Steerinp C 

Meeti.n(> Fo 

^lonsui 

' ■ C!^ 


wittee 

: SrJf-Studv __v 


3.0 


3.1 


3.2 '''ork On ^ 
"State Of The 
Art Report" 


3.3 Work On 
"State Of The 
Art Report" 


<2^<Work On 
"state Of The' 
Art Report" 


c?5Make 
Pevision's On 
First Report 


Meet With 
Air Force Group 
On Performance- 
Based Instructic 


n. ■ 


• # 


JJ) Work On 

"State Of The 
Art Report" 

rj-untia ^rpjecL, 


SO 'fal<e Pre- 
lim. Visit To 
The Mississippi 
Divn, of Voc. 
Education. 


Si "Make Prelim 

^ Visit To The' 

Louisiana Divn. 

of ^'ocational 

Education. 
• 




DECEMBER 1 2' 
3*4 5 6 7 8 9 
10 11 12 13 14 15 16 
17 18 19 20 21 22 23 
24 25 26 2/ 23 29 30 
31- 

9 


. , ■/ 1 '2 3 
i 5 6 7 8 9 10 
11 12 13 14 15 16 17 
13 19 20 21 22 23 24 
25 26 27 23 ' 



Th« Oouglai Steworl Co. 

ERLC 
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m^m OF ACTIVITIES - BEN A. HIRST, JR. , ' A-I^ENDIX C , ', 





' MOHDAY ' 


TUESDAY 


lEDESDAY 


THUKDAY 

e , 


FfflDAY 


mmm 


JANUAIjy / 
1 2 3 4, 5 9 
7 a 3 10111213 

M 15 16 1? 13 19 20 
21 22 23 2^ 25 28 27 
28 29 30 31 


mm 

12 3 
•4 5 6 7 3.9 10 

11 12 13 1M5 16 17 
18 13 20 21.22 23 24 
25 26 27 28 29 30 31 




i 


t 

/work On 
Agreement Form 
For Ort^anizatlon 
Of the Consor- 
tium. 

1 


c2 Work On 
Agreement Form 
FnT nrpani7fltion 
Of The Consor- 
tium. ' 


3 

1 - 


'V 


5 Prelin.Visit 
To Virginia' 
Divn. of Voc. 
Education. 


6 Prelim. Visit 
To North Carol in< 
Divn. of Voc. 
Education. 


1 Work on 
1 "State of The 
Art" Report; 
Fla. Project. 


State Dept. 
of Education 
Dr. Giehls 
Tallahassee, 
Florida 


^Finish First 
Draft of "State 
nf fhp Arf" Rpnn 

Fla. Project. , 

* 




// 


/5 M On 

Finish Draft 
Of "Stat,e Of 
The^Art" report. 

LINCOLN'S BIRIHDAy 


« ^ 

f3 I\'ork On 

Finish Draft 
Of "State of the 
Art" Peport. * 
Ga. State, Prelim 
Visit;Ga. State 
Dept. of Educ. 


Mail thf 
"State of the 
Art" Report. 
Prelim.Visit To 

50. Car. Divn. c 
Voc. Education. 

51, VALENTINE'S DAY 


15 Prelim.Visit 
To Texas Divn, 
of Voc. Educ. 

f 


/^Trellm. Visi 
To Oklahoma 
Divn. nf Vor. 
Education. 


n 


IS ■ 

i 

\ 


(Cj U^Xn Work 
On Conferenc 

Design For 
fConsortlum 
'Contact 

Consultants 

PRESIDENFSDAy 


20 Prelim. 
! VljSlt to kh 

Divn. of Voc. 

Education. 


5/ SACS Staff 
. Meeting 10 .'OC 
A.M. 

Work on Agreent. 
Form fi Organ. 
Report, 


c2c2 Prelim. 

VlsiyTo 
Missouri Divn. 
of Voc. 
Education. 

WASHINGTON'S BlliTHOAY 


(25 Washington, 
D.C. To Meet Wit! 

fliiVrlpiihini Sfnff 

O.E. and the 
Aerospace Educat: 
Foundation. 


on 


25 


?.6 Work On , 

Materials Fo: 
The Conference. 


27 First Draft 
Of Organization; 
Structure & 


s,s 

1 


< 






Make Reservation; 
Other Arrange- 
ments. 


Apreenient Forms 


- — -) 

( 
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CALENDAR OF ACTIVITIES - BEN A. HIRST, JR. 



APPENDIX C 



FEeituARy 

1 2 3 

4 s 6 7 1 no 

II 12 13 U IS 16 17 

18 19 20 21 22 23 24 
25 26 27 28 



Double-Check, 
\ccoTOpdatlons 
:)ther Arrange- 
ments. 
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ERIC 



APRIL 

1 2 3 n'6 7 
8 9 101M213H 
IS 1617 18 13 20 21 
22 23 24,25 26 27 28 
29 30 



5 Work .on 
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APPENDIX D 



RECOMMENDATIONS RESULTING FROM THIS STUDY 



AO 

a? 

l-*- The State Educational Research and Development Program should 
develop a comprehensive plan for the dissemination of products 
developed by the program or by contractors with the program. 
This dissemination plan should be designed in such a way 
that it becomes an integral part of ekch project whl,ch require 
broad dissemination of the product. 

2, In-servlce and pre-service training programs should be designed 
which will insure proper preparation of instructional .and 
Supervisory personnel for the implementation and assessment 
of the effectiveness of performance-based curriculum material 
and criterion-ref erencc^d test exercises. 



3, A system should be developed which would incdrporiite a mearjs of 
revising and updating catalogs of performance objectives and 
criterion-ref erenced test exercises, ' 



All three of these recommendations could be coordinated with 
certain elements of each making up a comprehensive plan for 
dissemination, in-service and pre-service training, and revision 
and updating of material. 
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